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FOREWORD 


Congressional  scrutiny  of  RDT&E  services  has  resulted  in  certain 
reappraisals  of  long-held  concepts  and  practices  concerning  the  utiliza- 
tion of  military  research  products  in  the  human  resources  area.  In  1975 
and  1976  the  General  Accounting  Office  (GAO)  conducted  extensive  inquiries 
among  the  personnel  and  training  laboratories  of  the  Army,  Navy,  and 
Air  Force  to  determine  whether  and  to  what  extent  the  products  of  their 
research  were  being  utilized.  Human  resources  scientists  realized  almost 
immediately  that  considerable  cooperation  was  required  to  define  terms 
and  develop  methodology  in  order  to  examine  these  issues.  What  came  out 
of  these  cooperative  efforts  were  the  beginnings  of  some  better  understand- 
ings of  what  constituted  products  and  utilization  in  the  human  resources 
area. 

An  attempt  at  formalization  of  these  understandings  and  of  some  useful 
definitions  was  made  by  Dr.  Arthur  J.  Druc.ker  , President  of  Division 
19  (Military  Psychology)  of  the  American  Psychological  Association,  in 
the  period  1975  to  1976.  Dr.  Drucker  selected  as  the  subject  of  his 
Presidential  Address  the  timely  problems  of  conceptualizing,  defining, 
and  effecting  product  utilization,  with  the  intent  of  presenting  to  his 
colleagues  the  current  state-of-the-art  of  research  product  utilization. 

The  widespread  interest  and  response  generated  by  his  address  has  resulted 
in  its  presentation  as  the  present  ARI  research  problem  review. 


J.  E.  UHLANER. 
Technical  Director 


MILITARY  RESEARCH  PRODUCT  UTILIZATION 


BRIEF 


Requirement : 

To  survey  U. S.  military  research  product  utilization  and  implementa- 
tion in  human  resources  RDT&E  and  determine  whether  any  generalization  of 
concepts  and  practices  among  researchers  and  users  on  the  subject  of 
product  utilization  can  be  made. 


Procedure : 

A 16-item  open-ended  questionnaire  was  informally  circulated  among  79 
Department  of  Defense  users  and  military  sponsors  and  among  104  researchers 
in  the  Army,  Navy,  and  Air  Force,  universities,  and  contractor  firms. 
Responses  received  from  19  users  and  military  sponsors  and  from  48  re- 
searchers were  tabulated  and  otherwise  summarized  to  provide  a basis  for 
generalization. 


Findings: 

Researchers  and  users  differed  in  their  definitions  of  "utilization." 
Researchers  tended  to  define  utilization  in  terms  of  use  and  change,  users 
in  terms  of  cost  savings  and  improved  effectiveness.  Both  users  and 
researchers  agreed  that  research  products  consist  of  much  more  than  pub- 
lications and  reports.  They  generally  agreed  that  the  most  successful 
method  of  insuring  that  the  user  uses  the  product  properly  is  to  "stick 
around  and  see  that  he  does."  The  idea  of  a "utilization  batting  average" 
was  unpopular,  although  a few  respondents  suggested  simple  formulas  for 
responding  to  inquiries  from  Congressional  Committees  on  quantity  of 
research  utilized.  Utilization  is  considered  very  important  in  program 
evaluation,  particularly  for  Advanced  Development  products;  it  may  not 
be  important  or  relevant  for  Basic  Research  products.  There  was  general 
agreement  that  the  researcher  should  assume  partial  responsibility  for 
product  utilization,  officially  or  otherwise.  Cost-effectiveness  evalu- 
ation is  important  but  is  not  widespread  among  RDT&E  activities  in  the 
human  resources  area. 


Utilization  of  Findings: 

These  findings  provide  preliminary  information  for  budget  and  program 
managers  for  use  in  the  translation,  implementation,  and  tracing  of 
human  resources  military  research  products  into  operational  use  and 
during  that  use. 


MILITARY  RESEARCH  PRODUCT  UTILIZATION  1 


For  many  years  human  factors  researchers  in  the  military  R&D  community 
had  difficulty  getting  much  attention.  Now,  it  seems  almost  overnight, 
they  are  under  the  glare  of  spotlights  directed  by  Congress,  Secretaries 
of  the  various  services,  and  certainly  the  Defense  Department — all  showing 
keen  interest  in  R&D  products  of  the  human  resources  laboratories  and 
looking  for  evidence  that  the  research  they  have  been  underwriting  all 
these  years  has  led  to  products  and  discernible  payoff.  Congress  enlisted 
the  aid  of  GAO — the  Government  Accounting  Office — for  enlightenment  on 
research  utilization. 

I don't  know  exactly  what  answers  Congress  wanted,  but  one  of  them 
must  have  been  just  how  much  human  resources  research  leads  to  products. 

The  personnel  and  training  laboratories  received  inquiries  on  research 
product  utilization  to  which  many  of  them  had  problems  responding,  in  some 
cases  because  they  couldn't  agree  on  the  definition  of  "product,"  let 
alone  what  was  meant  by  "utilization"  of  that  product.  It  was  not  sur- 
prising to  learn  that  some  of  the  conclusions  drawn  on  utilization  were 
not  very  encouraging  for  human  resources  research  efforts.  For  example: 
Having  performed  a utilization  count  among  researcher-designated  users  of 
the  publications  of  one  R&D  laboratory  in  the  human  resources  area,  a GAO 
official  concluded  that  only  19%  of  the  users  of  "40  reviews,  reports, 
memoranda,  papers,  etc."  reported  utilizing  the  results.  To  GAO,  reports 
represented  the  principal  products  of  research;  they  also  felt  that  "an 
entire  [R&D]  project  should  be  application-oriented,"  and  "users  should 
be  intimately  involved  in  the  whole  process  from  selection  and  formulation 
of  the  problem  to  writing  up  the  results,  implementing  the  results,  and 
evaluating  that  implementation"  (Buesse  , 1976).  By  implication,  if  the 
user  was  not  using  the  product — that  review,  report,  memorandum,  or  paper — 
researcher  and  user  were  not  having  nearly  enough  dialogue  before  , during  , 
and  after  the  research. 

Nor  did  researchers  get  any  sympathy  from  the  Deputy  Director  for 
Research  and  Engineering  (DDR&E).  He  also  hit  at  the  erratic  interaction 
between  human  resources  laboratories  and  potential  users,  implying  that 
In  some  cases  even  the  value  of  the  best  contributions  was  diminished  by 
inadequate  coordination  with  users.  According  to  his  office,  the  labora- 
tories "are  contributing  significantly  in  some  areas  and  not  contributing 
in  others  either  through  their  own  fault  or  by  virtue  of  circumstances 
beyond  their  control.  In  other  cases  this  lack  of  appreciation  is  due  to 
Inadequate  product  marketing.  A better  marketing  effort  should  be  under- 
taken by  these  laboratories  to  assure  not  only  that  their  current  contri- 
butions are  used,  but  also  that  the  opportunities  for  making  such  con- 
tributions can  be  expanded"  (White  and  Taylor,  1976). 


‘Presented  as  the  Presidential  Address  to  the  Military  Psychology  Division 
(Division  19)  of  the  American  Psychological  Association  at  the  APA  annual 
meeting,  4 September  1976,  Washington,  D.  C. 


As  I talked  to  small  groups  of  Division  19  members,  I found  a great 
deal  of  enthusiasm  for  ray  suggestion  that  the  R&D  community  should  share 
experience  and  look  at  the  problem  of  utilization  together.  Dr.  Wallace 
Slnaiko  was  later  to  note:  "I  think  we  are  long  overdue  to  convene  a 
meeting  for  the  purposes  of  defining  a continuous  and  serious  effort  in 
the  utilization  area.  Let's  start  small  and  see  what  happens." 

I did  something  like  that.  I started  small  and  tentative.  I recalled 
a saying  from  the  good  old  days — my  good  old  days  , that  is — that  if  you 
can't  figure  out  a direct  approach  to  the  solving  of  a given  problem,  you 
should  appoint  a committee,  or  you  should  run  a survey.  Since,  I'm  told, 
no  one  has  ever  erected  a statue  to  a committee,  I ran  a survey.  (But 
you  really  can't  get  away  from  committees — in  view  of  the  nature  of  ray 
survey  sample  1 have  to  regard  ray  respondents  as  a huge  ad  hoc  committee. 
Eventually  67  people  voluntered  to  serve  on  this  committee,  and  my 
thanks  go  out  to  all  of  them.) 

As  I said  in  the  survey  transmittal  letter  , I hoped  for  some  generali- 
zations that  could  help  the  R&D  community  on  the  general  subject  of 
utilization  and  on  better  specific  ways  to  prove  we  were  spending  the 
government's  money  responsibly.  I sought  the  views  of  technical  directors, 
project  directors,  users,  and  sponsors.  I sent  out  183  questionnaires — 

79  to  users  and  sponsors,  104  to  researchers.  I had  responses  from  19 
users  and  48  researchers  (Table  1) — most  of  them  courteous  and  helpful. 

(Too  helpful  , if  somewhat  gratuitous  , and  also  too  late  was  the  response 
of  an  Army  user  who  blue-penciled  my  transmittal  letter  with  the  zeal  of 
a first-year  English  professor.)  Who  responded?  Let's  look  at  Tables 
2 and  3. 


Table  1 

QUESTIONS  AND  ANSWERS 
USERS  AND  RESEARCHERS 


Returned 

Mailed  (X  of  Mailed) 


Users 

79 

(43Z) 

19 

(24X) 

Researchers 

104 

(57Z) 

48 

(46Z) 

183  (100X) 


67  (37X) 


Table  2 


WHO  RESPONDED? 
(POSITION  IN  R&D  COMMUNITY) 

In-House  Research  Directors 
Program  or  Project  Directors 
Bench  Scientists 
Contract  or  Grant  Monitors 
Contractors  and  Grantees 
Military  Sponsors 
Trainers 

Other  Military  Users 


Table  3 

WHO  RESPONDED? 
(ORGANIZATION  AND  FUNCTION) 


Army  Researcher  19 
Army  User  11 
Navy  and  Coast  Guard  Researcher  12 
Navy  User  3 
Air  Force  Researcher  6 
Air  Force  User  5 
Contractor  and  Other  11 


67 


- 3 - 


18 

15 

4 

3 

8 

12 

1 

6 

67 
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A larger  proportion  of  researchers  than  users  responded  , the  re- 
searchers being  generally  at  upper  echelons  of  their  research  organiza- 
tions. In  both  cases  we  probably  see  significant  experience  with 
research  as  a necessary  basis  for  replies.  User  representatives  tended 
to  be  military  and  to  rotate  to  different  kinds  of  assignments.  The 
larger  number  of  Army  respondents  quite  frankly  reflects  the  larger 
number  of  Army  contacts  I had  at  hand. 

Beyond  those  figures  I was  not  much  interested  in  tabulating  the 
responses.  Mainly  I kept  the  responses  as  raw  data — in  some  cases  , 
raw  remarks — but  they  were  mostly  sincere  and  conscientious  , and  I 
believe  that  respondents  were  convinced  that  what  they  had  to  say  was 
important  to  understanding  the  research  process. 

While  most  respondents  implied  that  military  research  could  be 
improved  in  one  or  more  aspects  of  product  utilization,  user  and  re- 
searcher indulged  in  charges  and  countercharges  as  to  what  is  wrong  with 
research  and  the  utilization  of  its  products,  despite  the  growing 
frequency  and  duration  of  their  dialogues. 

One  Army  user  was  friend  and  critic:  "In-house  researchers  need 
training  as  consultants  if  they  ever  expect  to  make  progress  with  the 
decision-makers.  Too  many  act  as  if  they  are  in  an  ivory  tower  and 
shroud  themselves  in  buzzwords  and  then  wonder  why  their  funds,  etc. 
are  cut  and  their  products  aren't  used  or  appreciated." 

Another  Army  user,  on  the  other  hand,  jabbed  at  research  require- 
ments people.  He  said  , "Most  research  I have  seen  has  been  the  result 
of  someone  looking  for  a crutch  or  a polite  way  of  putting  off  ? hard 
decision. " 

Still  another  Army  user:  "If  utilization  within  the  Army  is  to  be 
improved  , reports  must  identify  specific  utilization  actions  and  must 
be  produced  faster  to  improve  timeliness.  All  sponsor-user  organiza- 
tions must  have  at  least  one  professionally  qualified  individual  within 
their  staffs  to  coordinate  the  research  and  to  direct  utilization  of 
research  products." 

And  a user  from  the  Air  Force:  "Too  much  military  research  is 
unsolicited  or  useless  for  operational  decisions.  This  is  what  gives 
it  a bad  name." 

An  Army  researcher  replied:  "Today  utilization  seems  to  drive  all 
R&D , but  utilization  also  means  the  user  needs  to  be  effective,  not 
just  the  researcher,"  while  a Navy  researcher  was  conciliatory:  "I 
think  there  is  a learning  curve  in  which  the  R&D  organization  and  the 
user  organization  learr  to  work  with  one  another." 

"But,"  retorts  an  Army  user,  "practically  every  research  product 
I have  ever  seen  recommended  further  research  or  study." 


4 

Dr.  Bernard  J.  Fine  and  Mr.  R.  P.  Thorpe,  Army  and  Navy  researchers, 
respectively,  go  to  considerable  lengths  to  identify  with  the  user. 

Dr.  Fine:  "It  is  one  thing  to  have  a user  come  to  you  with  a specific 
problem  and  work  with  you  toward  its  solution.  It  is  quite  another  to 
attempt  to  establish  a relationship  based  on  a somewhat  vague  promise 
that  the  research  results  may  ultimately  be  of  great  use.  It  may  be  an 
extremely  important  area,  worthy  of  the  user's  support,  but  in  the  Army, 
at  least,  very  few  of  the  users  are  so  oriented.  So  research  on,  say, 
human  behavior  under  stress,  goes  on,  yielding  sometimes  interesting  and 
potentially  useful  information  if  fully  exploited;  but  (it)  gets  lost 
in  the  annual  report  , etc.  , and  very  rarely  makes  it  to  the  doctrine 
level." 

Mr.  Thorpe:  "My  personal  belief  is  that  an  applied  R&D  organization 
exists  only  to  help  meet  the  objectives  of  its  parent  organization.  To 
do  this,  its  products  must  be  utilized;  if  the  "state-of-the-art"  is  also 
advanced,  so  much  the  better.  But  having  an  impact  on  the  parent  organi- 
zation should  be  the  paramount  objective." 

And  yet  I am  sure  military  users  will  agree  that  scientific  research 
is  necessary  and  will  eventually  be  used.  One  researcher  summarized: 

"You  hope  that  all  good  basic  research  will  eventually  be  used.  If  all 
(or  nearly  all)  of  your  basic  research  is  being  used  immediately,  you  are 
probably  not  taking  enough  chances.  Not  'lack  of  use'  but  'lack  of 
imagination,  etc.'  are  the  crimes  in  basic  research." 

From  these  data  we  can  see  there  is  substantial  difference  in  the 
postures  taken  by  researchers  and  users.  The  opinions  expressed  were 
real  and  heartfelt.  Researchers  and  users  have  different  conceptuali- 
zations, different  perceptions  and  definitions,  when  it  comes  to  utili- 
zation, implementation,  and  products.  Some  sorting  out  seemed  called 
for. 

What  are  or  what  should  be  the  prerogatives  of  the  researcher  on 
the  one  hand  and  the  user  on  the  other?  Perhaps  the  questions  and 
answers  can  give  a clue.  I asked  , near  the  start  of  my  questionnaire  , 
"What  do  you  mean  by  the  term  'utilization  of  research'  or  'utilization 
of  research  products'  in  the  context  of  human  resources  R&D?"  (Table  4). 
Ask  an  open-ended  question,  you  get  an  even  more  open-ended  answer. 

From  the  responses  of  the  46  researchers  I heard  from,  utilization 
means  changes  in  the  way  things  are  done.  They  insisted  that  research, 
even  basic  research  done  under  6.1  budget  money,  is  designed  to  have 
discernible  impact  on  the  world;  i.e.  , we  do  not  pursue  knowledge  for 
its  own  sake.  I have  listed  contrasting  key  words  and  phrases  of 
researchers  and  users,  but  I'll  give  a few  of  the  user  responses  in 
in  full: 


Table  4 


WHAT  DO  YOU  MEAN  BY  THE  TERM  'UTILIZATION  OF  RESEARCH'  OR 
'UTILIZATION  OF  RESEARCH  PRODUCTS'  IN  THE  CONTEXT 
OF  HUMAN  RESOURCES  R&D? 


Researchers 

Operational  Use  of  Product 

Change — Mission  Performance 

Incorporation  of  Product 

Application — Knowledge 

Adoption  of  Results 

Minimum  Translation 

Options  for  Developmental 
Solutions 


Users 

Improve  Effectiveness 
Cost-effective  Improvements 
Usable  Information 
Benefit  Decision-making 
Provide  User  Requirements 
Save  Money 

Application— Operational 
Evidence 


"Application  to  conduct  of  current  mission  activities  to 
achieve  improved  effectiveness"  (Army  user). 

"Application  of  research  to  cost-effective  improvements" 

(Air  Force  user). 

"Research  produces  usable  information  or  products  that 
a)  assist  with  'real-world'  problems,  b)  can  be  understood 
and/or  applied  by  uninitiated  researchers"  (Army  user). 

"How,  where,  and  why  a research  product  is  used  to  benefit 
decision-making  at  predetermined  go/no-go  point"  (Army  user). 

"Result  of  a research  effort  used  to  aid  in  the  decision 
process"  (Air  Force  user). 

"Measurement  to  judge  probability  for  success  and  satisfac- 
tion so  as  to  manage  day-to-day  decisions  better"  (Navy  user). 

"Use  of  research  products  to  provide  user  requirements  not 
obtainable  by  normal  study  process"  (Army  user).  A sophisticated 
response  , that. 

"Use  of  research  to  save  money"  (Air  Force  user). 

"Operational  evidence  of  application"  (Air  Force  user). 
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There  appears  to  be  substantial  agreement  between  user  and  researcher 
that  a research  product  is  designed  to  effect  a change.  Were  there 
differences  between  researcher  and  user?  I conducted  no  formal  content 
analysis,  but  some  differences  between  researcher  and  user  responses  did 
strike  me.  If  you'll  allow  me  my  subjective  factor  analysis — of  response 
nuance — researchers  talked  in  terms  of  implementation , use  , application  , 
change  of  procedure . policy.  Users  talked  in  terms  of  profit  or  gains , 
improved  effectiveness,  cost-effectiveness  and  money  saving . problem 
solving , aids  to  better  decision  making  and  management.  As  a researcher 
I found  this  contrast  very  illuminating.  As  Fine  and  Thorpe  intimated  , 
perhaps  researchers  should  take  more  notice  of  the  user's  view  than  they 
have. 

Let's  now  leave  utilization  and  talk  of  research  products,  if  we  can 
indeed  separate  the  two.  The  question  on  products — "Which  would,  in 
your  opinion,  qualify  as  Research  Products?" — gave  away  a little  of  the 
plot  , but  I wanted  to  determine  the  reaction  of  respondents  to  products 
they  had  possibly  never  considered  before.  This  mind-expanding  exercise 
is  summarized  in  Table  5. 

I found  a tendency  for  the  researcher  to  check  more  items  as  products 
than  did  the  user;  in  fact,  for  each  item,  a greater  percentage  of  re- 
searchers acknowledged  that  it  qualified  as  a research  product.  Most 
notable  differences  between  researcher  and  user  were  found  in  the  items 
of  training  literature,  handbook,  report  suitable  for  publication  in  a 
professional  journal,  any  project  report,  recommendation  for  change  in 
policy,  SOP,  policy  manual.  I am  forced  to  speculate  that  the  user 
does  not  give  enough  credit  to  research  in  helping  him  with  his  imple- 
mentation of  policy  changes  or  may  not  accept  that  the  researcher  makes 
a contribution  with  such  products  as  handbooks  and  manuals.  At  the 
same  time,  the  user  apparently  is  reluctant  to  regard  the  report  as  a 
product  but  almost  universally  accepts  findings  (data)  and  research 
tools  as  legitimate  products. 

Who  _i£5  the  user  of  our  products?  With  what  kinds  of  people  do 
researchers  need  to  interact  in  their  efforts  to  effect  the  change 
implied  by  utilization?  I asked  my  respondents:  "Who,  specifically, 
are  the  ultimate  users  of  research  products  in  your  sphere  of  operations?" 
In  general  , responses  followed  the  categories  shown  in  Table  6.  We  can 
see  that  the  user  community  is  a broad  one , ranging  from  troops  in  the 
field  to  policy  makers. 

In  the  attempt  to  focus  respondents  (both  users  and  researchers)  on 
actual  cases  , I asked  them  whether  they  had  been  involved  in  the  im- 
plementation of  a research  product  or  products.  Products  thus  selected 
by  respondents  included  training,  testing  and  evaluation  of  personnel, 
job  standards,  simulations,  human  engineering,  physiological  products, 
cultural  change  products.  All  in  all,  56  separate  products  were 
reported. 


Table  5 


WHICH  WOULD,  IN  YOUR  OPINION,  QUALIFY  AS  RESEARCH  PRODUCTS? 


Product 

Percent  Mentioning 
Total  User  Rscher 

Research  Findings  (Data) 

85 

84 

85 

Tests  and  Other  Measurement  Instruments 

84 

74 

88 

Research  Tool 

81 

79 

81 

Training  Program 

79 

74 

81 

Training  Device 

76 

74 

77 

Training  Literature  (Manuals,  Circ , Instructions) 

76 

53 

85 

Set  of  Guidelines 

75 

63 

79 

Recommendations  for  Change  in  Doctrine 

73 

63 

77 

New  Technological  Information 

73 

63 

77 

Handbook 

73 

47 

83 

Report  Suitable  for  Publication  in  Prof  Journal 

70 

47 

79 

Any  Project  Report 

69 

47 

79 

Data  Banks 

67 

63 

69 

Technical  Manual 

67 

58 

71 

Recommendations  for  Change  in  Policy 

66 

42 

75 

Job  Aid  , Information 

63 

47 

69 

SOP  (Standing  Operating  Procedure) 

46 

26 

54 

Policy  Manual 

45 

26 

52 

Research  Material  for  University  Level  Education 

42 

21 

50 

Making  the  Researcher  an  Expert 

39 

21 

46 
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Table  6 


ULTIMATE  USERS  OF  RESEARCH  PRODUCTS 

Systems  Designers  , Managers  , Engineers  , and  Operators 
General  Staff 
Training  Officials 

Social,  Health  and  Educational  Agencies 

Personnel  Officials 

Members  of  the  Scientific  Community 

Troops  in  the  Field 

Industry 

Operational  Test  Desginers 
Policy  Makers 
Decision  Makers 
Planners 


When  I asked  who  took  the  initiative  to  get  these  56  different  products 
implemented  , user  and  researcher  did  an  Alphonse-Gaston  act.  Thirteen 
researchers  and  seven  users  claimed  the  user  did,  five  users  and  six  re- 
searchers said  the  researchers  did  , and  fourteen  said  it  was  a joint 
activity.  I concluded  that,  when  it  got  down  to  cases,  researcher  and 
user  did  acknowledge  the  contribution  of  each  other  in  implementation. 

What  factors  motivated  the  user  to  want  the  product  in  the  first  place? 
The  obvious  answer  was  need  of  some  kind  , usually  an  operational  need — 
for  a training  program,  procurement  decision,  better  classification,  better 
goals  and  methods,  better  recruiting  tools,  evaluation  of  a course,  re- 
duction in  cost  of  flying  training,  convincing  the  soldiers  that  a device 
was  indeed  useful,  a method  of  training  small  units  in  tactics,  univer- 
sally accepted  noise  limits  to  use  in  designing  aircraft,  achieving  high 
first-round  hits  on  the  TOW  antitank  weapon  irrespective  of  range, 
improved  symbology  for  presentation  of  computer-assisted  displays  in 
tactical  data  systems,  changes  in  testing  policy,  compliance  with  new 
Government  regulations,  selection  of  midshipmen  in  such  a way  that  80 
percent  would  voluntarily  enroll  as  engineering  or  mathematics  or  science 


majors,  reduction  of  attrition  costs,  help  with  morale  and  job  dissatis- 
faction problems.  As  you  can  see,  the  variety  of  needs  as  expressed  is 
endless.  But  1 can  bring  you  no  data  on  the  validity  of  those  needs  or 
what  challenges  they  imply  for  the  researcher. 

"What  factors  motivated  the  user  to  use  the  product"  elicited  not 
quite  so  many  obvious  responses  (Table  7).  Grouping  and  summarizing  the 
responses  may  nave  hidden  one  thread  of  significance,  which  I call  "faith 
or  hope"  on  the  part  of  the  user  that  he's  getting  some  help.  Consider 
some  of  these  irdividual  comments  by  researchers: 


Table  7 

FACTORS  THAT  MOTIVATED  THE  USER  TO  USE  THE  PRODUCT 


Recognized  Worth  of  Product 

Directive  or  Policy  Decision 

Confirmatory  Evidence 

Simplicity  or  Feasibility  of  Product 

Persistent  Strong  Need  for  Product 

Implementation  Inherent  in  Research 

Post-Research  Reinforcement  Activity 

Confidence  in  Researchers  and  Scientific  Backing 

Government  Requirement  (i.e.,  In  the  Contract) 

Persuasion 

Miscellaneous — High  Favorable  Visibility  of  Research,  User 
Support  at  Grass  Roots,  Contractor  Claims,  User  Partici- 
pation in  Staffing 


"User  in  constant  state  of  'management  by  crisis,'  i.e., 
ready  to  give  product  a try." 

"Product  presented  in  form  permitting  adoption  or  adapta- 
tion." 
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"Because  the  results  were  so  dramatic  and  potentially 
Increased  the  effectiveness  of  current  artillery  weapons  so 
much,  the  user  could  do  nothing  else  but  use  the  results." 

"Once  a decision  had  been  made,  it  was  necessary  to 
follow  through — so  it  was  momentum." 

"New  data  better  than  old." 

"Persuasion  by  me." 

"Operational  tests  showed  that  the  model  did  what  the 
user  wanted  it  to  do." 

"Faced  with  high  cost  problem  and  convinced  that  tech- 
nology could  a^aist  in  reducing  cost." 


I had  hoped  to  get,  from  this  question,  some  idea  how  a product  can 
best  be  sold,  but,  at  least  as  the  researcher  views  it,  replies  went  all 
over  the  place.  There  are  probably  as  many  reasons  for  the  user  to  be 
motivated  to  use  a product  as  there  are  users — and  products. 

If  one  asks  the  question  of  utilization  from  a negative  point  of 
view,  as  GAO  did,  some  clearer  concepts  emerge.  GAO  addressed  the 
question  only  with  respect  to  reports  or  publication-type  products. 

"A  number  of  possible  causes  of  low  utilization  rate  . . . noted  by  users 
were:  lack  of  timeliness,  lack  of  specificity  of  recommendations,  lack 
of  relevance  to  user  problems,  and  too  narrow  a research  scope  .... 
Noted  by  the  researchers  were:  lack  of  continuity  of  interest  by  sponsors, 
lack  of  user  understanding  of  results,  improper  use  of  research  end 
results,  and  lack  of  funding  for  implementation"  (Buesse  , 1976). 

We  have  all  experienced  the  frustrations  of  trying  to  utilize 
products,  knowing  that  the  actual  point  of  utilization,  particularly  of 
6.3  (Advanced  Development)  products,  has  had  virtually  no  officially 
proper  money  , few  personnel  who  can  be  legitimately  assigned  to  utiliza- 
tion or  implementation  activities,  and  little  acknowledged  responsibility 
for  seeing  that  a product  gets  used.  Products  have  been  known  to  get 
into  the  correct  system  nevertheless,  but  generally  through  ad  hoc 
arrangements  spurred  by  someone's  conscience  and  with  funding  from  who- 
knows-where. 

I asked:  "Did  you  get  a user  to  use  the  product  properly?  What  did 
you  have  to  do?"  (Table  8). 
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Table  8 


HOW  USERS  WERE  INFLUENCED  TO  USE  PRODUCT  PROPERLY 


Active,  Post-Research  Activity  (Usually 

Training)  22 

Report,  Documentation,  Briefing  Helped  9 

User/Developer  Agreement  or  Mutual  Involvement  7 

Directive  6 

Vigilance  4 

Persistent  Persuasion  4 

No  , did  not  get  them  to  use  product  properly  4 

56 


One  respondent  replied:  "By  exposing  them  to  it."  I would  question 
whether  mere  exposure  is  enough,  except,  perhaps,  in  those  situations 
where  a user  is  waiting  impatiently  with  a "hope  of  success  with  it 
and  fear  of  failure  without  it"  attitude,  or  where  it's  a tool  with 
face  validity  such  as  REALTRAIN.  But  in  a climate  of  officer  assignment 
turbulence  when  no  one  can  recall  who  it  was  that  had  been  enthusiastic 
about  the  product,  or  where  a need  is  "overtaken  by  events,"  or  where 
the  implementing  change  is  really  going  to  upset  things,  or  where  a user 
must  now  do  something  he's  afraid  he  will  be  criticized  for  not  having 
done  earlier  as  part  of  his  job — in  such  cases,  users  may  have  to  be 
pressured  into  using  the  product  in  any  way,  let  alone  properly. 

If  you  will  once  more  accept  a frequency  count  from  my  67  respondents, 
Table  8 suggests  that  the  most  successful  way  to  get  the  user  to  use 
the  product  properly  is  to  stick  around  and  see  that  he  does,  particu- 
larly since,  in  our  business,  you  have  to  drive  out  the  old  ideas  with 
new  ideas.  It's  not  like  introducing  a new  tank,  where  in  all  likelihood 
you  won't  have  the  old  tank  around  except  as  an  antique.  You  may  have 
to  send  a man  or  a team  to  oversee  the  implementation  or  the  training 
procedure.  You  may  have  to  be  a policeman  or,  as  one  respondent  puts 
it,  a crusader:  "Make  him  a believer.  Ask  him  to  be  'change  agent'; 
give  nim  recognition  for  his  services;  encourage  him  to  state  any 
problems  he  may  have  in  implementation." 
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One  researcher  recommends  "sweet  talk."  Another  tops  that:  "Demon- 
strate, advise,  persuade,  interpret,  train,  retrain,  brief,  and  pray." 

In  some  cases  the  user  may  take  extensive  formal  steps  to  effect  optimal 
implementation  to  serve  his  political  Interests.  For  Implementing  the 
ASVAB  (Armed  Services  Vocational  Aptitude  Battery)  a 50-person  group  was 
organized  to  provide  management  with  policies  for  correct  use  of  the 
instrument  , as  well  as  a policy  committee  at  DOD  level  , the  Armed  Forces 
Vocational  Testing  Group  (AFVTG)  for  high  school  testing,  and  the  Armed 
Forces  Examining  and  Entrance  Stations  (AFEES)  for  testing  the  applicant 
for  enlistment. 

In  my  questionnaire  I had  asked  for  experiences  in  implementation. 

In  the  main,  respondents  emphasized  success  stories.  Back  in  1968,  Dr. 
William  McClelland  listed  the  reasons  "why  products  don't  get  implemented," 
sifted  from  among  his  organization's  successes,  by  way  of  a scientific 
confessional  (Table  9).  Presumably  this  situation  was  typical  of  imple- 
mentation practice  at  that  time.  Much  of  it  is  still  to  be  found.  Are 
we  communicating  the  operational  value  of  research  any  better  today?  Is 
the  product  available  when  projected?  Is  the  operational  or  adopting 
structure  the  same  as  when  the  research  began?  How  about  continued  sup- 
port for  the  change  at  high  echelons?  Did  anyone  remember  to  get  funds 
and  personnel  programmed  for  projected  changes?  Is  the  product  in 
operationally  usable  form?  Does  proper  doctrine  exist  to  enable  opera- 
tional capability?  Does  someone  still  have  to  sell  the  product? 

(Dr.  McClelland  confesses  that  at  one  time  his  organization  thought  it 
wasn't  the  researcher's  job  to  sell  the  product.) 


Table  9 

WHY  PRODUCTS  DON'T  GET  IMPLEMENTED 
(from  McClelland) 


Poor  Communication 

Lack  of  Timeliness 

Too  Many  Changes 

Lack  of  Strong  Command  Support 

Lack  of  Funds  and  Personnel 

Lack  of  Engineering  Capability 

Policy  Problem 

Insufficient  Salesmanship 
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If , from  what  I've  reported  so  far,  you  get  the  Impression  that  im- 
plementation is  a hlt-or-mlss  matter  , you  may  be  correct  , but  perhaps  it 
won't  be  thus  for  long.  A current  service  directive  requires  the  imple- 
menting agency  to  request  additional  assistance  from  the  developing 
agency  in  effecting  implementation.  There  are  two  if's — 1)  IT  the 
developing  agency  has  sufficient  manpower  available  and  2)  J_f  the  imple- 
menting agency  has  sufficient  funds.  Further,  when  the  research  effort 
has  produced  additional  findings  which  the  sponsor  thinks  have  service- 
wide application,  he  is  required,  with  assistance  from  the  developer, 
to  prepare  a statement  of  methods  by  which  the  findings  were  put  to  use, 
with  recommendations  for  gaining  acceptance  of  the  research  product  and 
for  promoting  the  new  procedures.  Such  a directive  puts  new  responsi- 
bility on  the  sponsor  for  a very  vital  role  in  the  implementation  process, 
and  in  the  R&D  life  cycle  system,  for  that  matter;  it  should  bring  the 
role  of  the  user  of  software  closer  to  that  of  the  user  of  hardware. 

I asked  quantitative  questions  about  problems  in  product  utilization 
because  everyone  seems  preoccupied  with  how  much  and  what  kinds  of 
research  should  lead  to  full  implementation.  In  the  questionnaire,  I 
asked:  "How  should  a research  organization  determine  its  utilization 
batting  average?" 

Many  respondents  hemmed  and  hawed.  Five  researchers  rebelled  out- 
right : 


1.  "A  batting  average  is  inappropriate  ....  The 
median  or  standard  product  may  be  more  appropriate." 

2.  "This  is  an  inappropriate  concept.  Caneralizability 
is  the  measure  of  knowledge." 

3.  "Inappropriate.  Return  on  investment  is  a better 
concept  ....  Counting  reports  is  not  related  to  productivity." 

4.  "Irrelevant  question!  The  nation  should  understand 
R&D  is  essential  to  national  survival  , very  risky  and  has 
few  genuine  payoffs,  and  is  expensive." 

5.  "I  don't  know." 

One  summation  of  the  problem  is  worth  repeating:  "Utilization  by 
other  researchers  could  possibly  be  measured  by  number  of  citations  of 
articles  in  professional  Journals.  But  utilization  in  development  is 
difficult  because  there  is  usually  no  clear  thread  or  pathway.  Research 
results  are  always  combined  with  other  findings  or  decisions,  and  their 
use  is  rarely  acknowledged." 

But  for  those  who  accepted  my  inquiries  on  scientific  faith  (or 
face)  , here  is  what  the  collective  response  appears  to  be  (Table  10). 
Seven  respondents  suggested  formulae  or  equations,  such  as  "actual 
utilization  of  R&D  products  as  a percentage  of  total  R&D  conducted," 
which  most  of  them  proposed  in  essence.  But  most  of  these  proponents 
raised  doubts  as  to  the  efficacy  of  their  own  proposals. 
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Table  10 


HOW  A RESEARCH  ORGANIZATION  SHOULD  DETERMINE  ITS 
UTILIZATION  BATTING  AVERAGE 


Freq  of  Mention 

Some  Count  of  Customer  Use  20 

Predetermine  Utilization  (and  hence  indices  of 

utilization)  12 

A.  Plan  Each  Research  Product  for  Utilization 

B.  Plan  Documentation  and  Evaluation  in  Advance 

Consider  Projects  in  6.1,  6.2,  6.3  Budget  Categories 

Separately  8 

A.  Different  Products  for  Different  RDT&E  Research 
Categories 

B.  For  6. 1 , Journal  Acceptance  Rate 

C.  Did  6.  2 move  to  6.  3? 

D.  Hard  to  do — for  6. 3 (Applied) 

Some  Formula  or  Equation  Involving  Users,  Importance  and 


Other  Variables  7 
Analyze  Products  Before  and  After  Implementation  4 
Amount  of  Repeat  Business  3 
Documented  Feedback  3 
Number  of  Reports  (with  reservations)  2 


Dollars  Returned  on  Research  Investment 


1 


My  own  reaction  to  the  question  of  how  a research  organization  should 
determine  its  batting  average  is  that  as  long  as  Congress  and  GAO  can 
ask  questions  implying  the  importance  of  some  index  of  utilization  quantity 
the  question  is  not  an  inappropriate  one  for  human  resources  research 
to  consider.  But  it  is  a most  difficult  one.  How  much  utilization  is 
adequate  is  not,  as  Dr.  Wallace  Sinaiko  intimates,  one  that  a Congress  or 
a GAO  should  be  allowed  to  establish — even  by  implication.  We  must  , of 
course  , consider  research  in  6. 1 (basic  research)  , 6. 2 (exploratory  develop 
ment)  , and  6. 3 (advanced  development)  budget  categories  separately.  I 
would  say,  in  6.3  programs,  if  research  proceeds  efficiently  and  correctly, 
all  products  should  be  utilized  or  there  should  be  readily  understandable 
and  acceptable  reasons  why  not.  In  6.2  programs,  a fair  percentage — 
arbitrarily,  50%  of  our  efforts — should  move  to  6.3  research.  If  more, 
than  50%  moves  to  6.3  programs,  the  research  organization  is  probably  not 
taking  enough  technical  risk  in  its  exploratory  efforts.  The  category 
most  difficult  to  evaluate  is  6. 1 research.  Perhaps  no  consensus  should 
be  expected,  as  the  organization  itself  is  probably  best  qualified  to 
evaluate  6. 1 projects  within  its  own  environment. 

One  quantitative  aspect  of  research  product  utilization  that  cannot 
be  ignored  is  cost-effectiveness  evaluation.  Cost-effectiveness  evalua- 
tion, it  appears,  is  the  name  of  a spectator  sport  most  enjoyed  by  the 
budget  makers  and  program  decision  makers  and  most  feared  by  the  researcher 
players.  It  is  expensive  and  time-consuming  and  demands  rigorous  quanti- 
fication on  the  criterion  side  when  criteria  are  readily  available — 
which  they  usually  are  not.  But  for  the  next  few  years  or  more  , cost- 
effectiveness  evaluation  may  be  the  most  important  or,  even,  the  only 
game  in  town. 

I asked:  "If  you  conducted  a follow-up  evaluation  of  the  effective- 
ness of  the  research  product  , how  was  it  done?"  "If  it  included  a 
cost-effectiveness  evaluation,  what  was  the  criterion  used?" 

Two  traditional  approaches  to  measurement  of  cost-effectiveness  have 
been  a)  to  examine  alternatives,  looking  for  that  which  gives  the 
greatest  benefit  for  a given  cost,  and  b)  to  examine  alternatives,  looking 
for  that  which  produces  the  required  level  of  benefits  at  the  lowest 
cost.  The  name  of  the  game  is  to  produce  objective  measures  of  user 
benefits.  What  I didn't  specifically  try  to  obtain  was  respondent  experi- 
ence in  determining  whether  cost  benefits  were  actually  realized.  Such 
cost -effectiveness  information  is  difficult  for  most  research  organizations 
to  get  first  hand,  although  at  least  one  effort  is  now  known  that 
attempts  to  set  forth  guidelines  for  cost-effectiveness  analysis  in  the 
training  and  education  area.  1 refer  to  a recent  report  by  the  Naval 
Personnel  R6D  Center  by  Doughty,  Stern  and  Thompson  (1976). 

In  my  inquiry,  only  23  reported  that  some  kind  of  follow-up  evalua- 
tion of  the  effectiveness  of  their  research  product  had  been  done  (Table 
11).  Even  fewer  described  a cost-effectiveness  evaluation  criterion. 

In  fairness,  I must  say  I had  not  asked  respondents  for  their  total  or 
best  experience  on  follow-up  evaluation  or  cost-effectiveness  evaluation — 
only  that  experience  pertaining  to  the  product  they  chose  to  discuss. 
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Table  11 


HOW  WAS  THE  FOLLOW-UP  EVALUATION  CONDUCTED? 

Times  Mentioned 

Re  searcher /User  Agreement  or  Continued  Contact  6 

Collection  of  Data  Bases  3 

Further  Research — Usually  Validation  3 

Evaluation  Form  in  Report  for  Readers/Users  3 

Monitoring  Engineering  and  Operational  Tests  of  the 

System  1 

Development  Manual  and  Validation  Procedure  (SQT)  1 

Criterion:  Developers  are  able  to  follow  the  procedures  1 

Before  and  After  Performance  Measurement  1 

Attitude  Survey  of  Trainees  1 


Table  11  suggests  that  follow-up  evaluation  is  most  prevalent  where 
researchers  and  users  have  agreed  upon  it  or  continue  to  be  in  touch  with 
one  another. 

Few  generalizations  are  possible  with  respect  to  the  criteria  used  in 
cost-effectiveness  evaluation.  Table.  12  presents  a few  suggested  from 
the  responses:  for  instance,  training  costs  produce  the  same  level  of 
performance.  Or  firing  accuracy  is  increased,  while  time  and  ammunition 
costs  remain  the  same  or  are  decreased. 


Table  12 

IF  COST  EFFECTIVENESS  EVALUATION  WAS  INCLUDED, 
WHAT  WAS  THE  CRITERION? 


Training  Costs/Performance 
Firing  Accuracy/Time/Ammo  Costs 
Research  Costs/Cost  Avoidance 
Accident  Rate/Cost  Savings 
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To  the  last  question  to  be  discussed  from  the  questionnaire  proper — 
"What  role.  If  any,  should  utilization  play  in  program  evaluation?" — 
there  was  a high  level  of  unanimity,  summarized  as  follows: 

1.  Utilization  is  very  important — 

Especially  for  6.3  or  applied  programs  (laboratory 
survival  might  well  depend  upon  it). 

But  also  important  for  6. 2 programs. 

Not  important  (and  maybe  not  even  desirable)  for 
6. 1 programs. 

2.  But  be  patient — utilization  may  take  several  years. 

3.  And  don't  overlook  the  importance  of  negative  findings. 

I will  sum  up  briefly  and  only  in  part:  My  original  intention,  ex- 
pressed in  the  memorandum  accompanying  my  questionnaire,  was  to  seek — 
for  human  resources  researchers — areas  of  consistency  in  their  dealings 
with  the  people  who  can  influence  R&D  budgets.  I asked: 

"Under  what  circumstances,  if  any,  is  the  research  report  a proper 
and  adequate  product  for  R&D  effort?"  1 think  the  views  I received  are 
fairly  consistent — the  consensus  seemed  to  be  "when  talking  6.  1 or , in 
many  cases,  6.2  research."  But  conversely,  the  research  report  would 
appear  to  be  inadequate  as  a sole  product  for  6.3  research.  Additionally, 
even  if  it's  an  artifactual  consequence  of  the  questionnaire  format, 
a list  of  all  possible  types  of  products  was  agreed  to  number  between 
1 and  20,  and  researchers  named  closer  to  10-20  types. 

"Should  the  researcher  take  any  responsibility  for  product  utiliza- 
tion?" The  consensus  here  appears  to  be  even  stronger — yes  , in  fact, 
some  kind  of  researcher  participation  is  a necessity  for  product 
utilization.  Additionally  it  would  appear  that  everyone — user  and 
researcher--is  conscious  of  need  for  utilization  if  the  research  process 
is  to  be  regarded  as  complete.  I will  go  further.  It  is  my  impression 
that  the  most  successful  utilization  involves  the  cooperative  effort 
of  the  researcher  and  user  to  establish  a mechanism  or  structure,  on  the 
order  of  the  ASVAB  structure,  for  effecting  the  changes  required  in  using 
a new  research  product.  Such  mechanisms  are  not  unknown  to  the  services. 

The  high-level  user  may  have  to  beg,  borrow,  or  steal  services  and 
resources — anything,  at  the  beginning.  At  the  very  least,  we  have,  in 
the  AFEES,  Reception  Stations,  and  Depots  for  Tests,  and  in  training 
development  activities  staffed  by  users,  some  mechanisms  for  effecting 
the  changes  a new  research  product  requires  to  make  life  for  the  user 
easier  than  before. 

Is  it  Important  to  differentiate  between  utilization  and  implementation7 
A few  respondents  thought  so.  It's  late,  but  I think  it's  about  time  to 
do  so.  To  me  . utilization  is  the  broader  term  and  can  refer  to  actions 
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taken  by  researchers  and  users,  and  can  be  true  of  6.1,  6.2,  and  6.3 
programs.  Implementation  implies  a function  generally  performed  by  users, 
generally  following  6.3  efforts,  and  generally  not  a formal  part  of  the 
R&D  life  cycle  as  we  know  it  in  human  resources  research  (although  R&D 
people  are  frequently  involved).  A familiar  and  classical  example  is  a 
personnel  test  validated  through  R&D  efforts  but,  ideally,  standardized 
through  implementation  procedures.  Or  , a prototype  training  course 
developed  through  R&D  effort  is  utilized  by  training  managers  in  the 
implementation  process,  i.  e.  , generalized  for  widespread  application 
or  implementation,  perhaps  in  a number  of  training  locations.  Or, 
researchers  develop  engagement  simulation  methodology  as  part  of  6.  1 
and/or  6.2  effort,  leading  to  a training  model,  which  is  then  utilized 
by  researchers  to  test  doctrinal  concepts  as  part  of  6. 3 research. 

"How  widespread  is  the  practice  of  cost-effectiveness  evaluation  of 
research  products?"  The  answer  here  has  to  be  "not  very,"  but  many 
respondents  intimated  this  practice  must  receive  much  more  attention, 
starting  immediately. 

In  everything  I've  discussed  it  has  been  very  clear  to  me  that 
improved  utilization  means  new  specialty  roles  for  life  cycle  management 
in  software.  In  fact  , Dr.  Bernard  Rimland  would  have  us  go  to  Congress 
and  say,  "our  valuable  badly  needed  end  products  aren't  being  properly 
utilized,  despite  our  efforts."  Considerable  additional  funding  and 
personnel  resources  will  be  needed  to  make  these  specialties  a reality. 
Until  these  requirements  are  met  more  fully,  I think  we  will  continue  to 
have  utilization  and  implementation  problems. 

Finally,  let  me  thank  respondents  for  their  conscientious  thought 
and  willingness  to  make  a contribution  to  this  important  problem. 
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